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(57) An information processor which includes a 
function of printing visual information, sound information 
and relational information relating the visual information 
to the sound information on a print sheet, the visual in- 
formation being comprised of document information, im- 
age information and graphic information, and a function 
of reproducing the sound information based on the re- 
lational information prinled on the print sheet, comprises 
a relational information input portion for inputting the re- 
lational information comprised of the standard identifi- 
cation code which is indicative of standard sound infor- 
mation related to the predetermined range of the visual 
information, and the incidental identification code which 
is indicative of incidental sound information to the stand- 
ard sound information, a converting portion for convert- 
ing the input sound information and relational informa- 
tion to two-dimensional bar code information, a printing 
portion for printing the visual information and the two- 
dimensional bar code information obtained by conver- 
sion on the print sheet, a reading portion for reading the 
visual information and the two-dimensional bar code in- 
formation which are printed, a display portion for selec- 
tively displaying the read visual information on a screen, 
a designating portion for partially designating the prede- 
termined range of the displayed visual information, a re- 
storing portion for restoring the two-dimensional bar 
code information to the sound information related to the 
designated visual information and the relational infor- 
mation, and a reproducing portion for reproducing the 
standard sound information and incidental sound infor- 
mation related to the visual information in pairs based 
on each identification code. 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention relates to an information 
processor having two-dimensional bar code processing 
function, and more particularly to an information proc- 
essor having two-dimensional bar code processing 
function which has conventionally been utilized for a 
word processor, a personal computer, a communication 
terminal, a POS terminal, a cash register and the like, 
wherein visual information comprised of document in- 
formation, image information and graphic information, 
and sound information converted to two-dimensional 
bar code are related to each other according to the re- 
lational information, printed together on a print sheet, 
and combined based on the relational information print- 
ed on the print sheet so thai information transmission is 
performed. 

2. Description of the Related Art 

Conventionally, the visual information such as doc- 
ument information, image information or the like has 
been often exchanged and transmitted by means of a 
print sheet or a floppy disk. According to an information 
processor such as a word processor, a personal com- 
puter or the like, generally, the visual information is print- 
ed on the print sheet, recorded on the floppy disk, trans- 
mitted by means of a facsimile, a telephone line or the 
like, or copied from the print sheet and the floppy disk. 

Furthermore, OCR technology for optically reading 
the document information inputted to the print sheet by 
handwriting and converting the same to character font 
(type) has been developed and widely used for electron- 
ic translation, information exchange and transmission. 
There are many problems such as the character recog- 
nition ratio and the like. These problems are greatly re- 
duced by conversion into coded information such as a 
bar code. The bar code is formed by one-dimensional 
arrangement of numerical data and can easily be rec- 
ognized by machines. For this reason, the bar code is 
currently utilized in every field. 

As a typical example of the bar code, Japanese 
Laid-open Patent No. Hei 5(1 993)-12288 l has proposed 
an information processor. 

According to the information processor described in 
Japanese Laid-Open Patent No. Hei 5(1 993)-1 2288, 
document information is converted into coding informa- 
tion such as JIS code orthe like, and bar code is created 
and printed on a recording sheet. II necessary, the re- 
cording sheet is read and inversely converted to restore 
to the document information. The document information 
thus obtained can be utilized. Also in the case where the 
recording sheet is copied or transmitted by a facsimile, 
the coding information is seldom degraded and informa- 



tion absence and errors are not caused often. 

However, since the information volume of the one- 
dimensional bar code which can be recorded on the print 
sheet or the like is small, the application is restricted. 
5 For example, the one-dimensional bar code is not suit- 
able for coding sound information. 

Accordingly, a print sheet on which information is 
expressed in two-dimensionai bar code (mesh pattern) 
has recently been contemplated in place of the one-di- 
to mensional bar code. 

The two-dimensional bar code represents binary 
digital data bits. For example, a print color "white" is rep- 
resented by "CP and a print color "black" is represented 
by "1 ". The small square (mesh) expressed by this metri- 
cs od is arranged two-dimensionally, and printed and read 
so that the digital data which is recorded can be ob- 
tained. Consequently, the quantity of information which 
can be recorded is increased as compared with the one- 
dimensional bar code according to the prior art. If the 
20 sound information is recorded on such a print sheet in 
two-dimensional bar code having a mesh pattern, the 
quantity of the sound information which can be recorded 
is smaller as compared with the case where a magnetic 
tape is used. However, the sound information can be 
25 exchanged and transmitted by a facsimile and distribu- 
tion costs can be reduced. In addition, since the sound 
information cannot be decoded visually, secrecy can be 
observed. 

In general, a record medium such as a magnetic 
30 tape or a CD has been used for the sound information. 
In place of such a record medium, a print sheet on which 
the sound information is recorded in two-dimensional 
bar code having a mesh pattern has been contemplated 
(see Japanese Laid-Open Patent No. Hei 3(1991)- 
35 11482). 

Japanese Laid-Open Patent No. Hei 3(1 991 )-1 1 482 
has disclosed a voice recording method and device and 
a voice reproducing method and device in which a voice 
is digitized to generate digital data, and the generated 

40 digital data is converted to an image having a mesh pat- 
tern and printed on a print medium, and the print medium 
on which the meshpatterned image having voice infor- 
mation is printed is read by an image reader, digitized 
and analog-converted so that the voice is reproduced. 

45 According to Japanese Laid-Open Patent No. Hei 
3(1 991 )-1 1482, however, a paper is substituted for a 
magnetic tape as a record medium for sound informa- 
tion so that the characteristics of the paper as the record 
medium can be utilized only for recording and reproduc- 

50 tion. For example, the sound information is printed on 
the print sheet in two-dimensional bar code individually 
but is not related to other information. For this reason, 
the range of utilization of the sound information which is 
expressed in two-dimensional bar code is restricted. 

55 

SUMMARY OF THE INVENTION 

In consideration of the above circumstances, it is an 
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object of the present invention to provide an information 
processor having two-dimensional bar code processing 
function in which the standard sound information and 
incidental sound information related to visual informa- 
tion can be converted to two-dimensional bar code, the s 
two-dimensional bar code obtained by conversion can 
be printed on a print sheet, and the standard sound in- 
formation and incidental sound information printed on 
the print sheet can be reproduced in pairs. 

The present invention provides an information proc- 
essor having two-dimensional bar code processing 
function, which includes a function of printing visual in- 
formation, sound information and relational information 
relating the visual information to the sound information 
on a print sheet, the visual information being comprised 
of document information, image information and graphic 
information, and a function of reproducing the sound in- 
formation based on the relational information printed on 
the print sheet, comprising a visual information input 
portion for inputting the visual information, a sound in- 
formation input portion for inputting the sound informa- 
tion, a relational information input portion for inputting 
the relational information comprised of the standard 
identification code which is indicative of standard sound 
information related to the predetermined range of the 
visual information, and the incidental identification code 
which is indicative of incidental sound information to the 
standard sound information, a converting portion for 
converting the input sound information and relational in- 
formation into two-dimensional barcode information ac- 
cording to the predetermined procedure for conversion, 
a printing portion for printing the visual information and 
the two-dimensional bar code information obtained by 
conversion on the print sheet, a reading portion for read- 
ing the visual information and the two-dimensional bar 
code information which are printed on the print sheet, a 
display portion for selectively displaying the read visual 
information on a screen, a designating portion lor par- 
tially designating the predetermined range of the visual 
information which are displayed on the screen, a restor- 
ing portion for restoring the two-dimensional bar code 
information to the sound information related to the des- 
ignated visual information and the relational information 
according to the predetermined procedure for restora- 
tion, and a reproducing portion for identifying the stand- 
ard sound information and the incidental sound informa- 
tion based on each identification code of the relational 
information, and for reproducing the standard sound in- 
formation and the incidental sound information accord- 
ing to the predetermined procedure for reproduction. 

It is another object of the present invention to pro- 
vide an information processor having two-dimensional 
bar code processing function for synthesizing and re- 
producing the standard sound information and inciden- 
tal sound information related to the visual information at 
an arbitrary timing. 

It is yet another object of the present invention to 
provide an information processor having two-dimen- 



sional bar code processing function in which the. .visual 
information as well as the sound information can be con- 
verted into two-dimensional bar code, and the two-di- 
mensional bar code obtained by conversion can be re- 
lated to the visual information which is within the prede- 
termined range printed on a print sheet so as to be uti- 
lized. 

These and other objects, features, aspects and ad- 
vantages of the present invention will become more ap- 
parent from the following detailed description of the 
present invention when taken in conjunction with the ac- 
companying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram showing an information 
processor having two-dimensional bar code 
processing function according to an embodiment of 
the present invention; 

Figure 2 is a diagram showing a print sheet (1) to 
which two-dimensional bar code is applied accord- 
ing to the present invention; 

Figure 3 is a flowchart showing the processing op- 
eration according to Embodiment 1 of the present 
invention; 

Figure 4 is a flowchart showing the processing op- 
eration according to Embodiment 2 of the present 
invention; 

Figure 5 is a flowchart showing the processing op- 
eration according to Embodiment 3 of the present 
invention; 

Figures 6A and 6B are flowcharts showing a- sub- 
routine for each processing shown in Figure 5; 
Figure 7 is a diagram showing a print sheet (2) to 
which two-dimensional bar code is applied accord- 
ing to the present invention; 
Figure 8 is a diagram showing the designating op- 
eration for reproducing the two-dimensional bar 
code shown in Figure 7; 

Figure 9 is a diagram showing the reproduction and 
output processing of the two-dimensional bar code 
shown in Figure 7; 

Figure 10 is a diagram showing a print sheet (3) to 
which two-dimensional bar code is applied accord- 
ing to the present invention; 

Figure 11 is a flowchart showing the processing op- 
eration according to Embodiment .4 of the present 
invention; 

Figures 12A, 12B and 12C are flowcharts showing 
a subroutine for each processing shown in Figure 
11; 

Figure 13 is a diagram showing an example of the 
display of a list which manages two-dimensional bar 
code information; 

Figure 14 is a flowchart showing the processing op- 
eration according to Embodiment 5 of the present 
invention; 

Figure 15 is a diagram showing a print sheet (4) to 
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which two-dimensional bar code is applied accord- 
ing to the present invention; 
Figure 16 is a diagram showing an example ol the 
display of image information which is modified; 
Figure 1 7 is a flowchart showing the processing op- 
eration according to Embodiment 6 of the present 
invention; 

Figure 1 8 is a diagram showing a screen image ob- 
tained during the work for relating two-dimensional 
bar code to the arbitrary range of a document; and 
Figure 1 9 is a flowchart showing the processing op- 
eration according to Embodiment 7 of the present 
invention. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

The present invention provides an information proc- 
essor having two-dimensional bar code processing 
function which includes the function of relating visual in- 
formation comprised of document information, image in- 
formation and graphic information to sound information 
according to relational information and printing the same 
on a print sheet, and the function of reproducing the 
sound information based on the relational information 
printed on the print sheet, comprising a visual informa- 
tion input portion for inputting the visual information, a 
sound information input portion for inputting the sound 
information, a relational information input portion for in- 
putting the relational information comprised of the 
standard identification code which is indicative of stand- 
ard sound information related to the predetermined 
range of the visual information, and the incidental iden- 
tification code which is indicative of incidental sound in- 
formation to the standard sound information, a convert- 
ing portion for converting the input sound information 
and relational information to two-dimensional bar code 
information according to the predetermined procedure 
for conversion, a printing portion for printing the visual 
Information and the two-dimensional bar code informa- 
tion obtained by conversion on the print sheet, a reading 
portion for reading the visual information and the two- 
dimensional bar code information which are printed on 
the print sheet, a display portion for selectively display- 
ing the read visual information on a screen, a designat- 
ing portion for partially designating the predetermined 
range of the visual information which are displayed on 
the screen, a restoring portion for restoring the two-di- 
mensional bar code information to the sound informa- 
tion related to the designated visual information and the 
relational information according to the predetermined 
procedure for restoration, and a reproducing portion for 
identifying the standard sound information and the inci- 
dental sound information based on each identification 
code of the relational information, and for reproducing 
the standard sound information and the incidental sound 
information according to the predetermined procedure 
for reproduction. 



Accordingly, two or more sound information can be 
related to the visual information within the predeter- 
mined range of the print sheet and can be reproduced. 
For example, the standard sound information and the 
s incidental sound information can automatically be repro- 
duced in pairs. 

According to the present invention, the visual infor- 
mation input portion, the relational information input por- 
tion, the designating portion, the list information setting 
10 portion and the range setting portion are formed by an 
input device such as a keyboard, a pointing device (a 
mouse, a pen, etc.) or the like. 

The sound information input portion is formed by a 
microphone. 

is The converting portion, the restoring portion, the re- 
producing portion, the sound information synthesizing 
portion, the list information storage portion, the display 
control portion and the relational information creating 
portion are formed by a microcomputer comprised of a 

20 CPU, a ROM, a RAM and an I/O port. 

In particular the RAM is used as the sound infor- 
mation synthesizing portion and the list information stor- 
age portion, and as means for storing the input visual 
information and sound information. 

2B The ROM stores various application programs for 
two-dimensional bar code according to the present in- 
vention and programs for functioning as each portion 
which are executed by the CPU. The converting portion 
is formed by an A/D converter and a digital data/bar 

30 code converter. The restoring portion is formed by a bar 
code/digital data converter and a D/A converter. 

The printing portion is formed by a printer such as 
a thermal printer or a laser printer. The reading portion 
is formed by a scanner comprised of a CCD (charge 

35 coupled device) or the like, and is used as a device for 
inputting image information as well as reading the two- 
dimensional bar code information described on the print 
sheet. 

The display portion is formed by a display device 
40 comprised of a LCD (liquid crystal display), a CRT dis- 
play or the like. 

The voice output portion of the reproducing portion 
is formed by a speaker. 

According to the present invention, it is preferred 
45 that the information processor having two-dimensional 
bar code processing function further comprises a sound 
information synthesizing portion for synthesizing plural 
sound information based on the relational information. 

The relational information includes timing informa- 
so tion for causing the sound information synthesizing por- 
tion to synthesize plural sound information. Conse- 
quently, the plural sound information recorded in two- 
dimensional bar code can be reproduced at an arbitrary 
timing. Therefore, the plural sound information can be 
55 overlapped and reproduced. 

Preferably, the converting portion has the function 
of converting visual information to two-dimensional bar 
code information according to the predetermined proce- 
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dure for conversion, and the restoring portion has the 
function of restoring the two-dimensional bar code infor- 
mation to the visual information. 

According to the above structure, not only the sound 
information but also the visual information comprised of 
character information, image information and graphic in- 
formation are dealt with as the information related to the 
predetermined range. Consequently, the representation 
form of a document is expanded so that various using 
methods can be enhanced. 

It is preferred that the relational information includes 
reproduction condition information indicative of condi- 
tions on which the reproducing portion reproduces plural 
sound information. 

According to the above structure, the time condi- 
tions are set as reproduction condition information so 
that different information can automatically be selected 
and reproduced depending on the reproduction time (for 
example, the sound information about the greetings of 
morning, afternoon and evening). 

Preferably, the information processor having two- 
dimensional bar code processing function further com- 
prises a list information setting portion for setting list in- 
formation indicative of the attribute, coordinates/size 
and comment of plural two-dimensional bar code infor- 
mation which are related to the visual information, a list 
information storage portion for storing the set list infor- 
mation, and a display control portion for displaying the 
list information on the display portion in a multi-window. 

According to the above structure, the two-dimen- 
sional bar code information described on the print sheet 
can be listed and the contents thereof can be verified 
and managed simultaneously. Consequently, the two- 
dimensional bar code information can be edited more 
easily. 

It is preferred that the relational information includes 
position information indicative of the predetermined 
range of visual information and the range of two-dimen- 
sional bar code information related to the predetermined 
range. 

Prelerably, the information processor having two- 
dimensional bar code processing function further com- 
prises a modifying portion for modifying the information 
restored from the two-dimensional bar code information 
based on the position information of the relational infor- 
mation. 

According to the above structure, it is possible to 
verify whether the contents of the two-dimensional bar 
code information corresponds to the information which 
is within the related range or not, and to modify the 
same. 

The present invention will be described in detail with 
reference to the drawings and should not be construed 
as being limited by the following embodiments. An in- 
formation processor having two-dimensional bar code 
processing function according to the present invention 
is particularly applied to a word processor, a personal 
computer, a communication terminal, a POS terminal, a 



cash register and the like, and can relate sound infor- 
mation converted into two-dimensional bar code and the 
like to visual information such as document information, 
image information, graphic information and the like to 
5 be recorded on a print sheet, and can record and repro- 
duce them. 

Fig. 1 is a block diagram showing an information 
processor having two-dimensional barcode processing 
function according to an embodiment of the present in- 
vention. In Fig. 1, 1 denotes a keyboard which functions 
as a visual information input portion, a relational infor- 
mation input portion, a designating portion, a list infor- 
mation setting portion and a range setting portion, 2 de- 
notes a display device such as an LCD (liquid crystal 
display), a CRT display or the like which functions as a 
display portion, 3 denotes a pointing device for desig- 
nating coordinates on a display screen which functions 
as the designating portion and the range setting portion, 
4 denotes a coordinate detecting circuit for detecting po- 
sitional coordinates designated by the pointing device 
3, and 5 denotes a memory formed by a RAM which 
functions as a visual information storage portion, a two- 
dimensional barcode information storage portion and a 
list information storage portion. Various data which are 
being converted and restored are temporarily stored in 
work areas of the memory 5. 6 denotes an external stor- 
age device such as a floppy disk, a hard disk or the like, 
and is used as an auxiliary storage device. 

7 denotes a sheet reader which is a scanner com- 
prised of a CCD (charge coupled device), and functions 
as a reading portion for reading the visual information, 
two-dimensional bar code information and relational in- 
formation which are printed on the print sheet. In addi- 
tion, the sheet reader 7 is used as a device for inputting 
image information as well as reading an Information 
source printed on the print sheet. 

8 denotes a bar code/digital data converter for con- 
verting the read two-dimensional bar code information 
to digital data, and functions as a restoring portion. 9 
denotes a digital data/bar code converter for converting 
the digitized sound information to two-dimensional bar 
code, and functions as a converting portion. 10 denotes 
a printer such as a thermal printer, a laser printer or the 
like, and functions as a printing portion. 

11 denotes a microphone for fetching sound infor- 
mation and functions as a sound information input por- 
tion. 

1 2 denotes an A/D converter for converting the an- 
alog signal of the input sound information to a digital sig- 

so nal, and functions as a part of the converting portion. 

1 3 denotes a D/A converter for converting the digi- 
tized sound information to an analog signal, and func- 
tions as a part of the restoring portion. 

14 denotes a speaker for outputting the sound in- 
55 formation converted to the analog signal, and functions 

as a portion for reproducing the sound information. 

15 denotes an information processor for controlling 
all the circuits (each portion), which is formed by a mi- 
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crbcomputer comprised of a CPU, a RAM, a ROM and 
an I/O port. The ROM stores a program for controlling 
each circuit (each portion), various application pro- 
grams for two-dimensional bar code according to the 
present invention, and the like. The information proces- 
sor 15 functions as the converting portion, the restoring 
portion, the reproducing portion, the sound information 
synthesizing portion, the display control portion, the 
modifying portion, the list information setting portion, the 
'relational information creating portion and the like. 

The sheet reader 7 reads the sheet information 
printed on a print sheet. Then, the bar code/digital data 
converter 8 extracts two-dimensional bar code informa- 
tion from the read sheet information (sheet image) on 
the print sheet and converts the same to digital data. 

Various data such as the read sheet image, the two- 
dimensional bar code converted to the digital data and 
the like are stored in the memory 5. The sheet image 
and others are expanded into display data and dis- 
played on Ihe screen ol the display device 2. 

A user designates an arbitrary position of the sheet 
image displayed on the screen by the cursor operation 
of the keyboard 1 , the pointing device 3 or the like. The 
coordinate detecting circuit 4 detects the designated po- 
sition on a sheet. The information processor 15 decides 
whether the position is within the range to which two- 
dimensional bar code is related. If any, the related data 
are reproduced. 

The sound information which should be reproduced 
is outputted to the speaker 1 4 through the D/A converter 
1 3. The visual information comprised of character infor- 
mation, image information and graphic information 
which should be restored and reproduced are outputted 
to the display device 2. If necessary, a voice is letched 
by the microphone 11 , converted to digital data through 
the A/D converter 12, and stored in the memory 5. Var- 
ious data thus converted can be registered in the exter- 
nal storage device 6 and read out as required. 

In order to convert the related sound information 
and the like to two-dimensional bar code information 
and print the same on the print sheet, the digital data 
(for example, sound information) converted by the A/D 
converter 1 2 is converted to a two-dimensional bar code 
image through the digital data/bar code converter 9. 
Other visual information such as document information, 
image information and the like are directly outputted to 
the printer 10 and printed on the print sheet. 

Fig. 2 is a diagram showing a print sheet (1 ) to which 
two-dimensional bar code is applied according to the 
present invention. In Fig. 2, 30 denotes a print sheet on 
which visual information comprised of typed character 
information and image information, two-dimensional bar 
code information, and the relational information are 
printed together. 

As will be described below, the relational informa- 
tion which should be printed on the print sheet is divided 
into relative relational information and mutual relational 
information. The relative relational information includes 



position information which indicates the pbsition (coor- 
dinates) and area of the two-dimensional bar code in- 
formation and the visual information related to the two- 
dimensional bar code information. The mutual relational 
5 information includes management information for relat- 
ing and processing plural two-dimensional bar code in- 
formation. 

Field a describes the relative relational information 
converted by the two-dimensional bar code information. 

io Field b describes the typed character information. Field 
c describes the image information. Fields d1 to d4 de- 
scribe the two-dimensional bar code information. Field 
d1 describes the mutual relational information. Field d2 
describes the standard sound information. Fields d3 and 

is d4 describe the incidental sound information (depend- 
ent sound information). 

[Embodiment 1] 

20 in Fig. 2, if another sound information (d3) should 
be reproduced subsequently to the standard sound in- 
formation (d2), the identification code ol the standard 
sound information and the number (identification code) 
of the incidental sound information which should be re- 
25 produced successively are added to the beginning of the 
standard sound information. If the number is recog- 
nized, there is the sound information which should be 
reproduced successively. 

Fig. 3 is a flowchart showing the processing opera- 
30 tion according to Embodiment 1 of the present invention. 
In Fig. 3, a sheet image (sheet information) printed on 
a print sheet (see Fig. 2) is read from the sheet reader 
7 (Step S101), and the read sheet image is displayed 
on the display device 2 (Step SI 02). When first reading 
35 the sheet image, the information processor 1 5 recogniz- 
es the position on the print sheet for the two-dimensional 
bar code information of the required sound information 
from the relative relational information and the mutual 
relational information. 
40 The above information is recorded in the memory 5 
provided in a body and is managed by the identification 
code or number. If the memory 5 has enough room, the 
sound information converted to two-dimensional bar 
code is stored in the memory 5 provided in the body, and 
45 utilized as required. 

The arbitrary position of the sheet image displayed 
on the display device 2 is designated by the cursor op- 
eration of the keyboard 1 or pointing device 3 (Step 
S103). It is decided whether the designated position is 
50 within the related range or not (Step S104). If the des- 
ignated position is not within the related range, the rou- 
tine returns to the display screen of Step S103 and 
stands by. If the designated position is within the related 
range, each data for the related two-dimensional bar 
55 code is read (Step S1 05). It is decided whether the data 
is the standard sound information (standard voice data) 
or not (Step S106). The sound information is outputted 
to the speaker 1 4 through the D/A converter 1 3 so as to 
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be reproduced if it is not the standard sound information 
(Step S110). 

It is decided whether two-dimensional bar code in- 
formation corresponding to the number of incidental 
sound information (incidental voice data) is present or 
not (Step S1 07). If the corresponding number is present, 
the incidental sound information is read (Step S109). If 
the incidental sound information is not present, sound 
information is newly inputted from the "microphone 11 
and stored as incidental sound information in the mem- 
ory 5 through the A/D converter 12 (Step S108). 

The original standard sound information is repro- 
duced (Step S1 1 1 ). When finishing reproduction, the in- 
cidental sound information is reproduced successively 
(Step S112). The routine returns to the processing for 
reproducing the original standard sound information and 
incidental sound information and repeats the same as 
required (Step S1 13). 

According to Embodiment 1 shown in Fig. 3 : two or 
more sound information can be reproduced continuous- 
ly. If the first standard sound information is reproduced, 
the next incidental sound information is retrieved. If the 
prior sound information is completely reproduced, the 
next dependent sound information can also be repro- 
duced automatically. 

If the next sound information which is specified is 
not present, a voice is inputted from the microphone 11 
and reproduced as the next sound information. Accord- 
ingly, the standard sound information and the sound in- 
formation dependent thereon can be reproduced in 
pairs. 

For example, when hearing two sound information 
(voice data) in comparison with each other, an operator 
should designate reproduction of each sound. However, 
operation can be performed easily by the structure in 
which the two sound information can be reproduced 
continuously. 

When comparing a voice of a good example with 
your pronunciation in the conversation practice of for- 
eign languages, it is possible to alternately hear the 
voice of a good example and your pronunciation easily. 
Consequently, your pronunciation can be judged objec- 
tively so that a linguistic skill can be enhanced. 

[Embodiment 2] 

Fig. 4 is a flowchart showing the processing opera- 
tion according to Embodiment 2 of the present invention. 
Embodiment 2 shown in Fig. 4 will be described below 
with reference to the diagram for explaining the print 
sheet shown in Fig. 2. 

As shown in Fig. 2, the relative relational informa- 
tion expressed in two-dimensional bar code is printed 
on a portion of the print sheet (1 ) which is first scanned. 
The relative relational information includes the identifi- 
cation code, number and attribute of two-dimensional 
bar code information, and position information indicative 
of the range on the print sheet to which information is 
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related. Subsequently, visual information such as doc- 
ument information, image information and the like, and 
two-dimensional bar code information of mutual rela- 
tional information, sound information and the like are 

5 printed as sheet images (sheet information) on the print 
sheet. Consequently, these information are read by the 
sheet reader 7 and properly stored in the memory 5 
(Step S201 ). The read sheet image is displayed on the 
display device 2 (Step S202). 

10 The arbitrary position of the sheet image displayed 
on the display screen is designated by the pointing de- 
vice 3 (Step S203). It is decided whether the arbitrary 
position is within the related range or not (Step S204). 
If the arbitrary position is within the related range, the 

75 related two-dimensional bar code is read (Step S205). 
It is decided whether the data is the mutual relational 
information or not (Step S206). If the data is not the mu- 
tual relational information, the routine proceeds to the 
processing for reproducing the sound information (Step 

20 S21 1 ) so as to output the sound information (voice data) 
from the speaker 14. 

If the data is the mutual relational information, a 
work area for processing the voice data is reserved 
(Step S207). The two-dimensional barcode of the voice 

2S data is read according to the contents of description of 
the mutual relational information (Step S208). The voice 
data are synthesized according to the contents of de- 
scription of the mutual relational information in the work 
area of the memory 5 (Step S209). 

30 if the description of the mutual relational information 
is not ended (Step S210), the routine returns to the 
processing for reading the voice data. When the voice 
data are completely synthesized based on the contents 
of description of the mutual relational information, the 

35 synthesized voice data is outputted to the speaker 14 
through the D/A converter 13 so that voice data is re- 
produced (Step S211). 

The print sheet (1 ) shown in Fig. 2 will be described 
again. For example, character information such as a 

40 word of blessing for a Iriend who will get married is de- 
scribed in Field b on the print sheet. A message related 
to the sound information, i.e., "Congratulations!" is de-. 
scribed within the range b1 "(X2, Y2) - (X3, Y3)'. 

Two-dimensional bar code information and sound 

45 information thereof are printed in Fields a and dl to d4. 
The position information of the two-dimensional bar 
code information (d1 , d2, d3, d4), and character infor- 
malion (b1) and image information (c1) related to the 
two-dimensional bar code information are printed in two- 

50 dimensional bar code in Field a. 

The two-dimensional bar code of data 1 (d1 ) is the 
mutual relational information for relating and processing 
the two-dimensional bar code of plural sound informa- 
tion. Herein , the two-dimensional bar code of data 2 (d2) 

55 and data 3 (d3) which are sound information are related 
to each other. Timing information is set in such a manner 
that the sound information expressed in the two-dimen- 
sional bar code of the data 3 can be reproduced a little 
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earlier than the sound information expressed in two-di- 
mensional bar code of the data 2 

For example, the. mutual relational information for 
relating the data 2 (d2) to the data 3 (d3) is recorded in 
a first (d1 ) two-dimensional bar code information area. s 

The message of blessing "Congratulations!- is re- 
corded in a voice in a second (d2) two-dimensional bar 
code information area. . 

The sound of "Wedding march" is recorded in a third 
(d3) two-dimensional bar code information area. 10 

The data structure of two-dimensional bar code in- 
formation will be described below. The data will be given 
in hexadecimal numbers. Actually, however, the data is 
expressed in two-dimensional bar code. The numeric 
values which will be given do not always match the ac- is 
tual data recording method. A mechanism for error de- 
tection and the like are also omitted. 



.Data 

Data number : [01 00] 

Mutual relational information : [00 00], (0 : Mutual 
relational information, 1 : Voice data) 
Data 1, : [00 01 00 08 7F OA FF 08] 

The first two-dimensional bar code data (mutual re- 
lational information) is printed within the range of (X0 = 
0100, Y0 = 0800) - (X1 = 0A7F, Y1 = 08FF). 
Sheet image : [40 06 00 03 38 08 7F 03] 

When a cursor is within the range of (X2 = 0640, Y2 
= 0300) - (X3 = 0838, Y3 = 037F) of the sheet image, 
the first two-dimensional bar code data is read and re- 
stored. 

Separation code : [FF FF] 
Data number : [02 00] 

Voice data : [01 00], (0 : Mutual relational informa- 
tion, 1 : Voice data) 
Data 2 : [00 01 00 90 7F OA FF OA] 

The second two-dimensional bar code data (voice 
data) is printed within the range of (X4 = 0100, Y4 = 
0900) - (X5 = 0A7F, Y5 = 0AFF). 

Separation code : [FF FF] ... Some relative rela- 
tional information are arranged. 

Two separation codes [FF FF] which are arranged 
indicate that the relative relational information is ended. 
Subsequently, the character information, the image in- 
formation, and the two-dimensional bar code data of the 
mutual relational information and the sound information 
are printed. 

.Data 

Data [02 00, 00 00, 00 00 00 00, 03 00, 02 00, 60 
F0 FF FF] 

[02 00] : The second two-dimensional bar code data 
(voice data) is selected. 



[00 00]: No.O is set as a standard voice data 
number which is an identification code of the start 
of reproduction of the second voice data. (No.O 
voice data is virtual working voice data for dealing 
with plural voice data.) 

[00 00 00 00] : An offset value from the identification 
code of the No.O voice data which is the standard 
voice data is 0. 

[03 00] : The third two-dimensional bar code data 
(voice data) is selected. 

[02 00] : The second voice data is set to the stand- 
ard voice data of No. 3 voice data. 
[60 F0 FF FF] : The reproduction of the voice data 
is started earlier by 0.5 sec than No.2 voice data. 
(If a reproduction rate is 8000 data/sec, a hexadec- 
imal number FFFFF060 is a decimal number 
-4000.) 

At a party after a wedding, a friend reads, by means 
20 of a sheet reader, a print sheet on which two-dimension- 
al bar code is printed, and designates "Congratulations! 
M in the words of blessing which are displayed. Conse- 
quently, the friend's affectionate voice "Congratulations! 
u can be heard with "Wedding march" played. 
25 in addition, one of piano sounds is recorded in two- 

dimensional bar code. The waveform of the sound is 
modulated, or the amplitude of the sound is increased 
or decreased to perform processing. If the processed 
waveforms are synthesized at an arbitrary timing, the 
30 music can be played. 

In that case, if all voice data are synthesized in the 
work area of the memory 5 and output to the speaker 
14 as shown in the flowchart of Fig. 4, the required ca- 
pacity of the memory 5 of the information processor is 
35 greatly increased. 

In consideration of the case where one-second 
voice data is reproduced every 10 sees 100 times, the 
one-second voice data is 8K bytes and the size of the 
mutual relational information is 8 bytes for the descrip- 
40 tion of voice data synthesis, i.e., 1 K byte or less for 100 
voice data, for example. 

These two come to about 9K bytes. Consequently, 
these voice data can be printed as two-dimensional bar 
code information on the sheet. However, if the voice 
45 waveforms are synthesized without any device, 8 x 
((1 00' - 1 ) x 1 0 + 1 ) = 7928K bytes are needed. The in- 
formation processor which is currently used can keep a 
memory capacity of a little less than 8M bytes, but is 
very inefficient. 
so . Accordingly, only the data in the necessary part of 
each voice data are read and synthesized, and output- 
ted to the speaker 14 little by little, for example, every 
second to perform data synthesis for the next one sec- 
ond, and then the synthesized data are sequentially out- 
ss putted. Consequently, about 2 seconds are enough for 
the CPU of the information processor to handle the 
memory. 

Actually, plural voice waveforms are synthesized 



BNSDOCID:<EP 074361 4A2_ I _> 



8 



15 



EP0 743 614 A2 



16 



and outputled in real time so that very few work areas 
are required. Since plural voice data can be overlapped 
at an arbitrary timing, the voice can be reproduced for 
a very long time as compared with the information vol- 
ume printed on the sheet. 

According to Embodiment 2 shown in Fig. 4, it is 
possible to add the information which can arrange so 
that the plural sound information may be overlappingly 
reproduced at an arbitrary timing. The music can be 
played depending on a timing at which the voice data is 
processed and reproduced. 

[Embodiment 3] 

Fig. 5 is a flowchart showing the processing opera- 
tion according to Embodiment 3 of the present invention. 
In Fig. 5, a sheet image on a print sheet is read (Step 
S301), and the read sheet image is outputted and dis- 
played on the display device (Step S302). The displayed 
image is designated by the pointing device (Step S303). 
It is decided whether the designated position is within 
the related range or not (Step S304). 

If the designated position is not within the related 
range, the routine returns to the display screen desig- 
nation (Stop S303). If the designated position is within 
the related range, the related data is read (Step S305). 
It is decided whether the data is the mutual relational 
information or not (Step S306). If the data is not the mu- 
tual relational information, it is merely reproduced (Step 
S307). If the data is the mutual relational information, its 
header is verified so that the processing corresponding 
to the header is executed (Steps S308 to S310). 

The processing operation of Steps S301 to S306 is 
the same as that of Steps S201 to S206 shown in Fig. 4. 

Figs. 6A and 6B are flowcharts showing the subrou- 
tine for each processing shown in Fig. 5. Fig. 6A shows 
the subroutine of Step S308 in Fig. 5 (plural voice 
processing). The processing operation of Steps S308-1 
to S308-4 shown in Fig. 6A is the same as that of Steps 
S208 to S21 1 shown in Fig. 4. Therefore, the description 
is omitted. 

Fig. 6B shows the subroutine of Step S309 in Fig. 
5 (multidata processing). If not only voice data but also 
character data and image data are related in Step S309, 
these data are read (Step S309-Tj, the character data 
and the image data are outputted to the display device 
2 (Step S309-2) and the voice data is reproduced 
through the speaker 1 4 (Step S309-3). 

The information which indicates the attribute of data 
converted to the two-dimensional bar code information 
is added to the relative relational information. The infor- 
mation related to the two-dimensional bar code other 
than the voice information is added to the mutual rela- 
tional information which is newly added in Embodiment 
2. An example in which information is newly added to 
the specific example described in Embodiment 2 will be 
shown below. 



Data 

Data number: [01 00] 

Data attribute : [0900], (0 : Data, 1 : Text, 2 : Image, 
5 3 : Voice, .... 9 : Mutual relational information ... and 

the like) 

Data 1 : [00 01 00 08 7F OA FF 08] 

The first two-dimensional bar code data (mutual re- 
10 lational information) is printed within the range of (X0 = 
0100, Y0 = 0800) - (X1 = 0A7F, Y1 = 08FF). 

Sheet image : [40 06 00 03 38 08 7F 03] - 
When a cursor is within the range of (X2 = 0640, Y2 
= 0300) - (X3 = 0838, Y3 = 037F) of the sheet image, 
is the data is reproduced. 

Separation code : [FF FF] 
Data number : [02 00] 
Data attribute : [03 00], (3 : Voice) 
20 Data 2 : [00 01 00 90 7F OA FF OA] 

The two-dimensional bar code data (voice data) is 
printed within the range ol (X4 = 0100, Y4 = 0900) - (X5 
= 0A7R Y5 = 0AFF). 
25 Separation code : [FF FF] ... Some relative rela- 

tional information are arranged. 

Two separation codes [FF FF] which are arranged 
indicate that the relative relational information is ended. 
Subsequently, the character image, the graphic im- 
30 age and the two-dimensional bar code of the mutual re- 
lational information and various information are printed. 

.Mutual Relational information 

35 [4D 55 33 49 43 20 20 20] : 

Identification code (8 bytes) : "MUSICtXD n 

If the above identification code is changed to the 
identification code for the reproduction related to the da- 
40 ta having other attribute such as "MULTlQQO subse- 
quent data meaning is changed. 

[10 00 00 00] : data size (4 bytes) : 
It is indicated that the data is 16 bytes. 
Data : [01 00 00 00 00 00 00 00 
02 00 01 00 60 F0 FF FF] 

(The same data as in Embodiment 2 is described.) 

The reproducing representation methods for vari- 
ous data are prepared and combined lor utilization. It is 
assumed that the identification code is M MULTlQ[H ". In 
this case, character information, sound information and 
image information are related to one another and out- 
putted, whose sheet is shown in Fig. 8. 

Fig. 7 is a diagram showing a print sheet (2) to which 
two-dimensional bar code is applied according to the 
present invention. Fig. 8 is a diagram showing the des- 
ignating operation for reproducing the two-dimensional 
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bar code shown in Fig. 7, in which the sheet image on 
the print sheet is read by the sheet reader 7 and specific 
character information is designated so that the informa- 
tion expressed in two-dimensional bar code which is re- 
lated to the specific character information is called and 
reproduced. 

Fig. 9 is a diagram showing the processing for re- 
producing and outputting the two-dimensional bar code 
shown in Fig. 7. 

In Fig. 7, the multi-data processing function (identi- 
fication code : "MULTlODD u ) is set in such a manner 
that the picture of a lion : the text lor explanation and the 
roar of the lion are reproduced. If the character of. "Lion" 
in the displayed sheet images is designated (see Fig. 
8), the picture (or photograph) of the lion is displayed as 
image information and the text of explanation "Lion : a 
.big carnivorous mammal which lives in Africa, belongs 
to the feline family and is characterized by the mane of 
an adult male" is displayed as character information. In 
. addition, the roar of the lion is reproduced as a voice 
through the speaker-14. Fig. 9 shows the state in which 
these data are reproduced. 

According to Embodiment 3 shown in Fig. 5, not on- 
ly the sound information but also the character informa- 
tion and the image information can be related and out- 
putted. 

If the character data and the image data other than 
a voice can be related to the image on the print sheet 
and the like, and reproduced, visual effects (the display 
of,an annotation, the related chart and the like) can be 
, obtained. 

[Embodiment 4] 

Fig. 10 is a diagram showing a print sheet (3) to 
which two-dimensional bar code is applied according to 
the present invention. In Fig. 10, a word of "greeting" is 
displayed in the text. In this portion, the selection of 
voice data ("greeting") of two-dimensional bar code in- 
formation to be reproduced depending on a time is re- 
lated to the mutual relational information. If "Good after- 
noon" is designated, the current time is checked. When 
the time is about noon, "Good afternoon" is reproduced. 

Fig. 11 is a flowchart showing the processing oper- 
ation according to Embodiment 4 of the present inven- 
tion. With reference to Fig. 1 1 , the flow of the processing 
operation will be described below by using the print 
sheet (3) shown in Fig. 10. 

A sheet is read (Step S401 ), and the read sheet im- 
age is outputted to the display device and displayed 
thereon (Step S402). A screen is designated by the 
pointing device (Step S403). The relative relational in- 
formation is retrieved, and the presence of the two-di- 
mensional bar code information related to the range of 
the relative relational information is retrieved (Step 
S404). If present, the two-dimensional bar code infor- 
mation is read (Step S405). It is decided whether the 
attribute of the two-dimensional bar code is the selective 



mutual relational information or hot (Step S406). If the 
attribute of the two-dimensional bar code is not the se- 
lective mutual relational information, the processing for 
data reproduction according to Embodiment 3 is execut- 
s ed (Step S407). 

If the attribute of the two-dimensional bar code is 
th e selective mutual relational information, the condi- 
tions for selection are checked (Step S408). The data 
to be reproduced is selected from plural two-dimension- 
io al bar code depending on the conditions, and processed 
(Steps S409 to S410). 

Figs. 1 2A, 1 2B and 1 2C are flowcharts showing the 
subroutine for each processing shown in Fig. 11. Fig. 
12A shows the subroutine of Step S409 shown in Fig. 
is 11 (the processing of morning greetings). Fig. 12B 
shows the subroutine of Step S410 shown in Fig. 11 (the 
processing of afternoon greetings). Fig. 12C shows the 
subroutine of Step S411 shown in Fig. 11 (the process- 
ing of evening greetings). 
20 According to an example shown in Fig. 11 , the lime 
is the selecting condition. If it is morning, the character 
information on the document of the print sheet is 
changed to "Good morning" and the voice data which is 
related and reproduced is changed to "Good morning". 
25 |f it is about noon, "Good morning" is changed to "Good 
afternoon". If it is after-evening, "Good morning" is 
changed to "Good evening". 

According to Embodiment 4 shown in Fig. 11, the 
reproduction conditions are added to the mutual rela- 
30 tional information on which the mutual relation of the 
two-dimensional bar code information is described. The 
two-dimensional bar code information can arbitrarily be 
selected from plural two-dimensional bar code informa- 
tion on the reproduction conditions, and can be repro- 
35 duced. 

Different data can be related to each other and re- 
produced on the reproduction conditions so that a plu- 
rality of documents can be created. 

40 [Embodiment 5] 

Fig. 1 3 is a diagram showing an example of the dis- 
play of a list which manages two-dimensional bar code 
information. In the case where only the data number, the 

45 data attribute and the size are managed, two-dimen- 
sional bar code information is unknown before repro- 
duction as shown in Fig. 13. Consequently, a comtr.t:it 
on the contents of the two-dimensional bar code infor- 
mation is incorporated into the relative relational infor- 

50 matbn, the mutual relational information or data itself, 
and is added to items to be displayed at the time of list- 
ing. 

Moreover, a sheet image that is within the range to 
which the two-dimensional bar code information is re- 
55 lated is displayed as required so that the two-dimension- 
al bar code information can be verified. When creating 
the relative relational information, extra blank space 
(such as right, lelt and bottom margin portions which are 
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set when printing) is set in advance around a portion on 
which the two-dimensional bar code information is prim- 
ed. 

It is determined that at least one related range, 
should be set to each two-dimensional bar code infor- 
mation, and that the designation of the pointing device 
to the sheet image is ignored with respect to the already 
related range. 

If it is determined that the above range can be rec- 
ognized as the related range only when listing, a com- 
ment can be referred to when adding some memoran- 
dum about the two-dimensional bar code information. 

An example of the mutual relational information da- 
ta to which the comment is added will be shown below. 
It is determined that the size of the comment is always 
arranged next to a header and that the comment corre- 
sponding to the size follows. 

[4D 55 53 49 43 20 20 20] : 
Header : "MUSICOQD" 
[11 00 00 00]: 

Comment size : A comment is 1 7 bytes.- 
[43 6F 6E 67 72 61 74 75 72 61 74 69 6F 6E 73 
2121]: 

Comment : "CONG R ATU LATIONS ! ! " 
(The above contents are displayed when listing 
the two-dimensional bar code.) 
[10 00 00 00] 

Data size : Data is 16 bytes. 
Data [01 00 00 00 00 00 00 00 ... ] follows. 

A sheet image on a print sheet on which the two- 
dimensional bar code information is printed is read, and 
the two-dimensional bar code information is listed. Ac- 
cording to Embodiment 5, the added comment and the 
sheet image which is within the related range are dis- 
played in addition to the information set to the two-di- 
mensional barcode (serial numbers, data attribute, data 
print range and data size) (see Fig. 13). operation ac- 
cording to Embodiment 5 of the present invention. In Fig. 
1 4, an area for managing list information is reserved and 
initialized (Step S501). The reading of the print sheet is 
started. First of all, the relative relational information is 
read and the position information of the two-dimensional 
bar code information on the print sheet or the like is ob- 
tained (Step S502). The two-dimensional bar code in- 
formation is read based on the relative -relational infor- 
mation (Step S503) and added to a list (Step S504). This 
work is repeated as long as the two-dimensional bar 
code information is present (Step S505). The list is dis- 
played (Step S506) and a processing branch is execut- 
ed according to user's operation (Step S507). By cursor 
operation or the like, the field of the two-dimensional bar 
code in the list is inverted to indicate that it is selected. 
When a "reproduce" button is pressed, the data is re- 
produced in the form corresponding to data attribute 
(Step S508). 

When a "register" button is pressed, the two-dimen- 
sional bar code data is stored in an external storage de- 
vice (Step S509). When a "print" button is pressed, the 



selected two-dimensional bar code is printed (Step 

5510) . When a "delete" button is pressed, the selected 
two-dimensional bar code is deleted from the list (Step 

5511) . When characters are inputted or changed in the 
5 comment field of each two-dimensional bar code, the 

comment is added to data or the contents of change are 
reflected (Step S512). When the additional reading 
function is designated, the routine returns to the 
processing for reading a new sheet (Step S502). If "f in- 
fo ish" is selected, the processing is ended. 

More specifically, it is almost impossible for a user 
to distinguish data by two-dimensional bar code. There- 
fore, the data which is listed is unknown before repro- 
duction. Even if the data are collected so as to be utilized 
75 later on, the memory for the data is lost after a while. 
After all, it is necessary to recognize the data again. A 
user who creates data or utilizes the data again can add 
a comment to the data and can readily verify the added 
contents, so that a data user can grasp the data easily. 
20 For example, assume that 5 kinds of two-dimen- 
sional bar code are printed on a sheet. When reading 
and listing the two-dimensional barcode, it is impossible 
to decide whether the data is object data or not if an 
annotation is not displayed but the attribute and data 
25 size are displayed. When sequentially reproducing and 
verifying the listed data, useless work should be execut- 
ed four times if fifth data is the object data. Such useless 
work can be avoided. 

According to Embodiment 5 shown in Fig. 14, the 
30 contents necessary for the management of the two-di- 
mensional bar code information are listed and dis- 
played. Consequently, the management contents of the 
two-dimensional bar code information can be distin- 
guished easily. 

35 

[Embodiment 6] 

The identification code indicative of data which is 
automatically executed based on the attribute of two- 

40 dimensional bar code information (for example, "FIRS- 
TEXE") is set. When first reading a sheet, the two-di- 
mensional bar code information having this identifica- 
tion code is retrieved as automatic execution two-di- 
mensional bar code (automatic execution data). A lay- 

45 out is changed or document images are modified ac- 
cording to the contents of the two-dimensional bar code 
information. 

Fig. 15 is a diagram showing a print sheet (4) to 
which two-dimensional bar code is applied according to 

50 the present invention. More specifically, Fig. 15 shows 
an image on the print sheet having the modifying func- 
tion which includes automatic execution data (modifying 
information). Fig. 16 is a diagram showing ah example 
of the display of image information which is modified. 

55 More specifically, Fig. 16 shows an example of the dis- 
play of image information which is modified based on 
the automatic execution data while the print sheet is 
read and displayed. 
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Fig. 1 7 is a flowchart showing the processing oper- 
ation according to Embodiment 6 of the present inven- 
tion. More specifically, Fig. 17 shows, in detail, the 
processing to be executed after the sheet is read and 
before the sheet image is displayed. 

Sheet reading is started (Step S601 ). While reading 
the sheet image, it is decided whether the relative rela- 
tional information is printed or not (Step S602). If the 
relative relational information is not printed, the process- 
ing is not required particularly and is ended. 

. If the relative relational information is printed, two- 
dimensional bar code is extracted from an image for the 
relative relational information and converted into digital 
data (Step S603). The relative relational information 
thus obtained is analyzed (Steps S604 to S608). Since 
plural two-dimensional barcode information are includ- 
ed, the attribute, the printed coordinates, the related 
range and the like are sequentially checked (Steps S604 
and S605), and the coordinates and range are stored in 
the memory 5 (Steps S606 and S607). This processing 
is repeated until ali the relative relational information ex- 
pressed in two-dimensional bar code are completely 
processed (Step S608). 

The sheet image reading is continued. The two-di- 
mensional bar code information is properly read from 
the sheet image according to the data of relative rela- 
tional information groups which have already been ob- 
tained, and stored in the memory 5 (Step S609). 

When the reading is finished, the contents of each 
two-dimensional bar code information are checked. If 
there is an error, processing is executed (Step S610). 
Each two-dimensional bar code information is checked 
so that it is decided whether the automatic execution da- 
ta is present or not (Step S611). If the automatic execu- 
tion data is present, the sheet image is changed and 
modified according to the automatic execution data 
(Step S61 2). The sheet image is outputted to the display 
device 2 and displayed thereon (Step S613). 

While plural two-dimensional bar code information 
are selected and reproduced in Embodiment 4 shown 
in Fig. 1 1 , the selected contents are not reflected until a 
user performs operation. If modification can be execut- 
ed before the sheet image obtained by reading the print 
sheet is displayed, it is possible to eliminate a problem 
that the display does not conform to the reproduced con- 
tents during operation. 

In order to keep the precision of character informa- 
tion and image information by changing the layout of the 
print sheet and reading the print sheet, the character 
information and the image information can be recorded 
as two-dimensional bar code information in advance 
and sheet images can be spread on the print sheet be- 
fore display. 

If a blank for a memorandum for two-dimensional 
bar code is painted out in white, the information images 
which are not necessary for writing to the print sheet are 
not displayed by the display device. 

According to Embodiment 6 shown in Fig. 17, the 



information on the print sheet which has been read is 
automatically modified before output, so that the infor- 
mation within the range to which the two-dimensional 
bar code is related can match the information which is 
5 related and reproduced. 

[Embodiment 7] 

Fig. 18 is a diagram showing a screen image ob- 

10 tained during the work for relating two-dimensional bar 
code to the arbitrary range of a document. 

Fig. 1 9 is a flowchart showing the processing oper- 
ation according to Embodiment 7 of the present inven- 
tion. More specifically, Fig. 19 shows the flow of the work 

is for relating the two-dimensional barcode lothe arbitrary 
range of the document. 

The data which should be arranged on the sheet is 
selected from a list (Step S701), and the related range 
is verified and set on the screen (Step S702). The infor- 

20 mation processor 1 5 decides whether the set range and 
the range which has already been arranged overlap 
each other or not (Step S703). If they do not overlap 
each other, the routine proceeds to the processing for 
deciding whether the next edit should be executed or 

25 not (Step S707). If they overlap each other (see Fig. 1 8), 
the two-dimensional bar code data which overlap are 
listed (Step S704). It is decided whether the mutual re- 
lational information which relates plural two-dimension- 
al bar code information should be created or not (Step 

30 S705). 

If the mutual relational information should be creat- 
ed, the information processor 15 prepares the informa- 
tion for relation such as the selection of the data repro- 
ducing method which is determined by arrangement po- 

35 sitions and the attribute of each data, and a user desig- 
nates a parameter which has not been set yet (Step 
S706). When the arrangement of two-dimensional bar 
code is finished, it is decided whether the next two-di- 
mensional bar code is arranged and edited or not (Step 

40 S707). If it is to.be continued, the routine returns to the 
processing for selecting two-dimensional bar code data 
(Step S701). If it is to be ended, the processing for ar- 
rangement and edit is finished. 

More specifically, it is verified whether the ranges 

45 to which plural two-dimensional bar code are related 
overlap or not, and the routine can proceed to a mode 
for automatically editing the mutual relational informa- 
tion. Consequently, a sheet creator does not need to 
create the mutual relational information in advance if 

so plural two-dimensional bar code should be related to the 
same portion. 

It is very hard to create the mutual relational infor- 
mation in consideration of the range where the two-di- 
mensional barcode is related and arranged. According- 

ss |y, data is actually arranged during the edit work for re- 
lating the two-dimensional bar code to the sheet. In the 
stage where various mutual relations are fixed, the mu- 
tual relational information is created as required. Con- 
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sequently, the work can be performed easily. In addition, 
the arrangement position and kind of the data are fixed 
so that the information processor can create most of in- 
formation automatically. Thus, the work can be per- 
formed more easily. $ 

According to Embodiment 7 shown in Fig. 19, the 
mutual relational information need not be created in ad- 
vance before the print sheet is edited if plural two-di- 
mensional bar code should be related to the information 
within the arbitrary range of the print sheet and repro- io 
duced. During the edit work for arranging the two-dimen- 
sional bar code information on the print sheet, the mu- 
tual relational information can be created and edited as 
required. 

According to the present invention, the following ef- is 
tects can be obtained. 

The visual information which is within the predeter- 
mined range on the print sheet and two or more sound 
information which are converted into two-dimensional 
bar code information are set so as to be related to each 
other and reproduced. For example, the standard sound 
information and the incidental sound information can au- 
tomatically be reproduced in pairs. 

Plural sound information expressed in two-dimen- 
sional bar code which are related to the visual informa- 
tion within the predetermined range on the print sheet 
can be reproduced at an arbitrary timing. Consequently, 
the plural sound information can be overlapped and re- 
produced Furthermore, if sound information is over- 
lapped and repeated plural times at an arbitrary timing, 
the sound information can be reproduced for a longer 
time as compared with the information volume de- 
scribed on the print sheet. In addition, the music can be 
played depending on the processing and reproducing 
liming of the sound information. 

An information source which is related to the pre- 
determined range on the print sheet includes the visual 
information such as character information, image infor- 
mation and the like as well as the sound information. 
Consequently, the representation form of a document is 
expanded so that more various two-dimensional bar 
code using methods can be enhanced. If the character 
information and the image information can be displayed 
on the screen, visual effects can be obtained. 

Reproduction conditions are added to the-' mutual 
relational information which indicates the mutual rela- 
tion between the two-dimensional bar code information, 
so thai the two-dimensional bar code information can 
be selected arbitrarily from plural two-dimensional bar 
code information and reproduced. 

Since the contents necessary for the management 
of the two-dimensional bar code information are listed 
and displayed., the management contents of the two-di- 
mensional bar code can be distinguished easily. 

When oulputting, to the screen, the information 
within the range to which the two-dimensional bar code 
information is related, the same information can be mod- 
ified automatically so as to match the information which 
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should be reproduced. 

In the case where plural two-dimensional bar code 
information should be related to the information within 
the arbitrary range of the print sheet and reproduced, 
the mutual relational information can be created and ed- 
ited as required during the work for arranging the two- 
dimensional barcode information on the print sheet and 
editing the same. 

Although the present invention has fully been de- 
scribed by way of example with reference to the accom- 
panying drawings, it is to be understood that various 
changes and modifications will be apparent to those 
skilled in the art. Therefore, unless such changes and 
modifications depart from the scope of the invention, 
they should be construed as being included therein. 

Claims 

1. An information processor having two-dimensional 
bar code processing function, which includes: 

a function of printing visual information, sound 
information and relational information relating 
the visual information to the sound information 
on a print sheet, the visual information being 
comprised of document information, image in- 
formation and graphic information, and 
a function of reproducing the sound information 
based on the relational information printed on 
the print sheet; 

the information processor comprising: 

a visual information input portion for inputting 

the visual information; 

a sound information input portion for inputting 
the sound information; 

a relational information input portion for input- 
ting the relational information comprised of the 
standard identification code which is indicative 
of standard sound information related to the 
predetermined range of the visual information, 
and the incidental identification code which is 
indicative of incidental sound information to the 
standard sound information; 
a converting portion for converting the input 
sound information and relational information in- 
to two-dimensional bar code information ac- 
cording to the predetermined procedure for 
conversion; 

a printing portion for printing the visual informa- 
tion and the two-dimensional bar code informa- 
tion obtained by conversion on the print sheet; 
a reading portion for reading the visual informa- 
tion and the two-dimensional bar code informa- 
tion which are printed on the print sheet; 
a display portion for selectively displaying the 
read visual information on a screen; 
a designating portion for partially designating 
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the predetermined range of the visual informa- 
tion which are displayed on the screen; 
a restoring portion for restoring the two-dimen- 
sional bar code information to the sound infor- 
. mation related to the designated visual informa- 
tion and the relational information according to 
the predetermined procedure for restoration; 
and . 

a reproducing portion for identifying the stand- 
ard sound information and the incidental sound 
information based on each identification code 
of the relational information, and for reproduc- 
ing the standard sound information and the in- 
cidental sound information according to the 
predetermined procedure for reproduction. 

2. The information processor having two-dimensional 
bar code processing function according to claim 1, 
further comprising a sound information synthesiz- 
ing portion for synthesizing plural sound information 
based on said relational information. 

3. The information processor having two-dimensional 
bar code processing function according to claim 2, 
wherein said relational information includes timing 
information for causing said sound information syn- 
thesizing portion to synthesize plural sound infor- 
mation. 

4. The information processor having two-dimensional 
bar code processing function according to claim 1, 
wherein said converting portion has the lunction of 
converting visual information to two-dimensional 
bar code information according to the predeter- 
mined procedure for conversion, and said restoring 
portion has the function of restoring the two-dimen- 
sional bar code information to the visual informa- 
tion. 

5. The information processor having two-dimensional 
bar code processing function according to claim 1 , 
wherein said relational information includes repro- 
duction condition information indicative of condi- 
tions on which said reproducing portion reproduces 
plural sound information. 

6. The information processor having two-dimensional 
bar code processing function according to claim 1, 
further comprising a list information setting portion 
for setting list information indicative of the attribute, 
coordinates/size and comment of plural two-dimen- 
sional bar code information which are related to said 
visual information, a list information storage portion 
for storing the set list information, and a display con- 
trol portion for displaying said list information on 
said display portion in a multi-window. 

7. The information processor having two-dimensional 



bar code processing function according to claim 1 , 
wherein said relational information includes modi- 
fying information for directing that the contents of 
the display of the visual information which is within 
s the range related to the two-dimensional bar code 
information should be modified. 

• 8. The information processor having two-dimensional 
bar code processing function according to claim 7, 

10 further comprising a modifying portion for modifying 
the contents of the visual information wh ich is within 
the range related to the two-dimensional bar code 
information to those of the two-dimensional bar 
code information which is prepared in advance by 

is the. modifying directions of said relational informa- 
tion. 

9. The information processor having two-dimensional 
bar code processing function according to claim 1 , 

20 further comprising a range setting portion for selling 
the range of visual information related to two-di- 
mensional bar code information, and a relational in- 
' formation creating portion for deciding whether the 
set ranges overlap each other or not, and addition- 

25 ally creating the identification code of the relational 
information which is within the overlapped range. 

10. The information processor having two-dimensional 
bar code processing function according to claim 1 , 

30 wherein said relational information includes posi- 
tion information indicative of the predetermined 
range of visual information and the range of two- 
dimensional bar code information related to said 
predetermined range. 
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FIG. 2 
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FIG. 3 
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FIG. 4 
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FIG. 6 
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FIG. 13 
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FIG. 14 
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FIG. 15 
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FIG. 17 
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